Method and arrangement for developing a three dimensional model of an environment

ABSTRACT

The present invention relates to a method and arrangement for developing a 3D model of an environment. The method comprises steps of providing a plurality of overlapping images of the environment, each image associated of navigation data, providing distance information, said LIDAR information comprising a distance value and navigation data from a plurality of distance measurements, and developing the 3D model based on the plurality of overlapping images and the distance information. The step of developing the 3D model comprises the steps of providing the 3D model based on the plurality of overlapping images; and updating the 3D model with the distance information using an iterative process.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/762,174, filed Jul. 20, 2015, published as US 2015/0362595, Dec. 17,2015, which is a U.S. National Stage of PCT Application No.PCT/SE2013/050044, filed Jan. 21, 2013, published as WO2014/112911, Jul.24, 2014, all of which are hereby incorporated by reference in theirentireties.

TECHNICAL FIELD

The present invention relates to a method and arrangement for developinga 3D model of an environment.

TECHNICAL BACKGROUND

A fast growing market both in civilian and military business isgeographical information systems. Knowledge about geographicalconditions forms a fundamental decision support to companies,authorities and in the military. The geographical information cancomprise digital maps having superposed information layers such asinfrastructure, terrain type and different types of objects. This way ofproviding digital maps is time consuming and comprises forming twodimensional maps comprising capturing images of the terrain from anaircraft and post-processing of the captured images. It is an even moretime consuming process to form three dimensional maps from capturedimages or range data sets of the terrain/infrastructure.

WO 2009/003529 relates to another type of geographical informationsystem. It relates to an arrangement and a method for providing a threedimensional map representation or a model of an area. The arrangementcomprises a processing unit arranged to, for a plurality of timerecorded, overlapping images of the area to be stereo processed,associate navigation states so that each pixel of each time recordedimage is correlated to a corresponding navigation state and to performthe stereo processing based on the associated navigation states so thatall pixels in the map representation or 3D model are specified in threegeographical dimensions.

One problem with the stereo processing of overlapping images is that asharp spatial change tends to disappear or partly disappear during thestereo processing since only a part of the overlapping images catchesthe sharp spatial change. One way to overcome this problem is tointroduce a laser rangefinder or a LIDAR device in the vicinity of theimaging device. Laser range measurements are conducted on a particularobject or area during the period when the overlapping images are takenof the particular object or area. Each laser range measurement is veryaccurate at a single point and can be used to improve the accuracy ofthe 3D model.

US2010/0204974 discloses a LIDAR and one or more electro-optical (EO)imaging device which may asynchronously acquire LIDAR shots and EOimages. Navigation and timing data may be used to associate a particularLIDAR shot and/or EO image with navigation data. The navigation data maybe used to cross correlate a LIDAR shot to a selected plurality ofoverlapping EO images. Ranging model information may be determined fromEO image sequences using a stereo imaging technique. The stereo imagingtechnique may be seeded using the LIDAR shot data.

One object of the present invention is to further improve modelling.

SUMMARY OF INVENTION

This has in one example been solved by means of a method for developinga 3D model of an environment. The method comprises the steps ofproviding a plurality of overlapping images of the environment, eachimage being associated to navigation data; providing distanceinformation, said distance information comprising a distance value andnavigation data from a plurality of distance measurements; anddeveloping the 3D model based on the plurality of overlapping images andthe distance information. The step of developing the 3D model comprisesthe steps of providing the 3D model based on the plurality ofoverlapping images and updating the 3D model with the distanceinformation using an iterative process.

In one option, the step of providing of a plurality of overlappingimages of the environment comprises capturing a plurality of overlappingimages of the environment using an imaging device, providing navigationdata related to the images; and associating the plurality of overlappingimages with respective navigation data.

In one option, the step of providing the distance information comprisestransmitting a plurality of pulses from a distance measuring device forreflection in the environment, receiving pulses reflected from theenvironment; providing navigation data related to the transmitted and/orreceived pulses; determining information related to a relation betweentime of transmission and time of reception of each received pulse; andassociating positioning data to each information related the relationbetween time of transmission and time of reception with respectivenavigation data;

The distance information may be provided by means of LIDAR, wherein thepulses are laser pulses. The distance information may be provided bymeans of radar. The distance information may be provided by means ofsonar.

The navigation data comprises information regarding position,orientation and timing.

In one option, the method further comprises a step of determining aweight relation between the distance measurement and the provided 3Dmodel, wherein the updating of the 3D model with the distanceinformation is based on the determined weight. The weight relation maybe determined based on a percentage of a difference between the valuefrom the distance measurement and the provided model. The weightrelation may be determined based on an uncertainty in the provided 3Dmodel. The weight relation may be determined based on an uncertainty inthe distance measurement.

In one option, the step of developing the 3D model comprises the stepsof:

a) determining stereo distances from the overlapping images;b) generating the 3D model based on the stereo distances from aselection of the overlapping images;c) determining the difference between the 3D model and the distanceinformation;d) updating the 3D model based on the difference between the 3D modeland the distance information and based on the determined weightrelation,e) evaluate the updated model against the provided model to determinewhich of the models is most accuratef) updating the selection of the overlapping images based on theevaluation,g) repeating step b) to f).

Step e) of evaluating the updated model against the provided model maycomprise re-projecting the stereo images based on the updated 3D model.

Step f) of updating the selection of overlapping images may compriseselecting only those images and/or sub-images which show parts of theenvironment relevant for the most accurate model.

Step g) of repeating the step of developing the 3D model may comprisethe steps of determining a difference between the model generated basedon stereo distances and a model generated based on stereo distances in aprevious step, wherein the development of the 3D model is exited if theif the difference is below a selected level.

The 3D model may be represented as a mesh. The 3D model may berepresented as a surface representation. The 3D model may be representedas a voxel representation

The invention also relates to a computer program comprising a programcode for developing a 3D model of an environment, comprising the step ofproviding a plurality of overlapping images of the environment, eachimage associated of navigation data, providing distance information,said LIDAR information comprising a distance value and navigation datafrom a plurality of distance measurements; and developing the 3D modelbased on the plurality of overlapping images and the distanceinformation. The step of developing the 3D model comprises the steps ofproviding the 3D model based on the plurality of overlapping images; andupdating the 3D model with the distance information using an iterativeprocess.

The invention also relates to a computer program product comprising aprogram code stored on a computer readable media for developing a 3Dmodel of an environment, comprising the step of providing a plurality ofoverlapping images of the environment, each image associated ofnavigation data, providing distance information, said LIDAR informationcomprising a distance value and navigation data from a plurality ofdistance measurements; and developing the 3D model based on theplurality of overlapping images and the distance information. The stepof developing the 3D model comprises the steps of providing the 3D modelbased on the plurality of overlapping images; and updating the 3D modelwith the distance information using an iterative process.

In one embodiment, the present invention comprises an arrangement fordeveloping a 3D model of an environment, said arrangement comprising amemory arranged to store a plurality of overlapping images of theenvironment, each image associated to navigation data and to storedistance information comprising a distance value and navigation datafrom a plurality of distance measurements; a processing unit arranged todevelop the 3D model based on the plurality of overlapping images andthe distance information. The processing unit is arranged to determinethe 3D model based on the plurality of overlapping images;

and to update the 3D model with the distance information using aniterative process.

BRIEF DESCRIPTION OF FIGURES

The invention will be further described with reference to theaccompanying drawings.

FIG. 1 illustrates an arrangement for developing a 3D model of anenvironment at least partly mounted on an aircraft.

FIG. 2 illustrates schematically the function of a camera in thearrangement in FIG. 1.

FIG. 3 illustrates schematically the function of a camera and a LIDARdevice in the arrangement in FIG. 1.

FIGS. 4a and 4b illustrate schematically an example of a scanningpattern provided by the LIDAR device in the arrangement in FIG. 1.

FIG. 5 is a block scheme illustrating an example of the arrangement ofFIG. 1.

FIG. 6 is a schematic illustration of a first part of modelling a scenebetween buildings.

FIG. 7 is a schematic illustration of a second part of modelling a scenebetween buildings.

FIG. 8 is a schematic illustration of a scene wherein a laser pulse istransmitted.

FIG. 9 is a schematic illustration of modelling a scene with a forest.

FIG. 10 shows a flow chart illustrating an example of a method fordeveloping a 3D model of an environment.

FIG. 11 shows a flow chart illustrating an example of a method fordeveloping a 3D model of an environment.

DETAILED DESCRIPTIONS OF INVENTION

In FIG. 1, an arrangement 101 for developing a three dimensional modelof an environment 103 is mounted on a movable carrier 102. In theillustrated example, the carrier is a in an airborne vehicle. Theairborne vehicle is for example a manned or unmanned fighter or civilianaircraft. The movable carrier is in an alternative example (not shown) asatellite, a land vehicle, or a watercraft, for example a lorry, ship orsubmarine. The arrangement 101 can also be hand held or mounted on aperson. In one example, only parts of the arrangement are mounted in themovable carrier while other parts, for example processing parts, are ata remote location.

The three dimensional model provided by the arrangement 101 is relatedto geographical coordinate system. The 3D model is provided from stereoimage processing a plurality of overlapping geo-referenced images of theenvironment. The geo-referenced images are provided from at least onecamera. In addition thereto, the 3D model is updated with informationfrom a distance measuring device.

In one example, one or a plurality of cameras can be used comprising acamera for visual light, an IR camera, and/or a video camera. Thedistance measuring device can be any type of distance measuring devicearranged to determine a distance with spatial resolution. The distancemeasuring device comprises a transmitter arranged to continuouslytransmit pulses a receiver arranged to receive pulses transmitted fromthe transmitter and reflected in the environment. The distance measuringdevice is arranged to determine a distance to the reflection point basedon the time difference between transmittal and reception at a certainpulse. The distance measuring device is in one example operating basedon optical light, ultrasonic and/or radar based. The optical distancemeasuring device is in one example LIDAR based. In one example, theoptical distance measuring device comprises a laser transmitter and adetector arranged to detect laser radiation. In one example, thetransmitter is a projector transmitting a light pattern and theassociated receiver is a camera. In the following description, thedistance measurements will be described in relation to LIDAR.

Thus, the arrangement for developing the three dimensional modelcomprises at least one camera and a LIDAR device supported by a movablecarrier 102. The LIDAR device is configured to obtain ranginginformation of the environment by transmitting laser energy towards theenvironment and detecting laser energy reflected and/or emitted therefrom.

The arrangement 101 for developing the three dimensional model is thenarranged to first provide the 3D model based on the plurality ofoverlapping images and then to update the 3D model with the LIDARinformation. A difference between the provided 3D model and the LIDARinformation is in one example determined at each location where LIDARinformation is available. The weight of the LIDAR measurement in theupdated model is determined based on a predetermined scheme. Forexample, the weight of the LIDAR information is determined as apercentage of the of a difference between the 3D model and the LIDARinformation. In one example, the percentage is 100% of the difference,i.e. the weight of the LIDAR information is 100%. In an alternativeexample, the weight of the LIDAR information is 40-60%. In one example,an uncertainty in the 3D model is determined at each location whereLIDAR information is available. In this example, the weight of the LIDARinformation is also or instead determined based on the uncertainty inthe 3D model. Detailed examples of uncertainties in the 3D model will bedescribed in relation to FIG. 5. Further, an uncertainty can also bedetermined related to the LIDAR measurement. The weight of the LIDARmeasurement in the updated model is then also or instead determinedbased on the uncertainty in the LIDAR measurement.

In FIG. 2, at least one camera 204 is illustrated supported by a notshown movable carrier. At a first time, the at least one camera ispointing in a first direction to a first field of view 205 a for a firstimage 206 a captured by the camera 204. At a second time, the at leastone camera 204 is directed in a second direction to a second field ofview 205 b for a second image 206 b captured by the camera 204. The atleast one camera 204 is arranged to provide a plurality of at leastpartly overlapping images 206 a, 206 b each covering at least a part ofthe environment. The model can be made better the more images fromdifferent camera positions are available in modelling an object or asurface in the environment. In one example, when images are capturedfrom an airborne vehicle, some surfaces visible from many positions inthe air are captured in 20 or more different images while other surfacesare visible in fewer different images.

In FIG. 3, a distance measuring device (not shown) is arranged on amoving carrier together with above described camera(s) 304. We hereindescribe the distance measurement device in relation to LIDAR. Theranging information provided from the LIDAR measurements is used toassist developing a 3D model. The LIDAR device is directed to the groundto scan with pulses 307 the same parts as covered by images 306 a, 306 btaken by the camera(s) 304. The area of a LIDAR pulse is much smallerthan the area of an image taken from the same height. Therefore in oneexample, a plurality of LIDAR pulses are provided within the area ofeach image.

In FIGS. 4a and 4b , examples of different scanning patterns 408, 409 onthe ground provided by a distance measuring device arranged on a movingcarrier are illustrated. The direction of the movable carrier (notshown) is illustrated with an arrow. In 4 a, the laser is arranged toperform a movement back and forward in a direction substantiallyperpendicular to the direction of movement of the movable carrier (notshown) to scan the environment in a zigzag pattern 408. In FIG. 4b , theLIDAR device is arranged so that the laser beam directed to theenvironment rotates in a circle, thereby providing LIDAR pulses 231forming a helical pattern 409 as the carrier moves in the direction ofthe arrow so as to scan the environment. In one example, the LIDARdevice is arranged to perform the linear and/or circular movement. In analternative example, the LIDAR device is fixedly arranged and a mirrorarrangement in the beam path from the LIDAR device is arranged toperform a pivoting movement so as to provide the linear and/or circularscanning. An advantage with LIDAR pulses forming a helical pattern isthat high resolution in all direction are achieved since the differencebetween two LIDAR pulses in a certain direction is small at leastbetween some pairs of LIDAR pulses along the plurality of LIDAR pulsesforming a helical pattern. It should be mentioned that the plurality ofLIDAR pulses can form any other pattern, such as sine formed pattern orany irregular pattern.

In one example, the camera images are taken and the distance measuresare performed at different times. In an alternative example, the cameraimages are taken and the distance measures are performed at the sametime.

In the example of FIG. 5, a 3D model is provided using an arrangement501 for developing a three dimensional model of an environment. Thearrangement 501 comprises at least one camera 504 arranged to generateimages. The at least one camera 504 is arranged to provide a pluralityoverlapping images covering the environment for which the model isbuilt. The camera is for example a camera for visual light or an IRcamera.

The arrangement 501 comprises further a distance measuring device 510.As described in relation to FIG. 1, the distance measuring device 510can be any type of distance measuring device arranged to determine adistance with spatial resolution. For example, ladar, sonar, distancemeasurement using structured light and/or radar can be used in additionto measurements based on camera images. As stated above, in thefollowing description, the distance measurements will be described inrelation to LIDAR.

The arrangement 501 comprises in accordance with this example further apositioning system 511 or a receiver of a positioning system arranged toprovide positioning and direction information related to the at leastone camera and related to the LIDAR device. The direction informationrelates to the optical direction of the camera/LIDAR device. The imagesare associated to this positioning and direction information. Further,the distance measurements are associated to this positioning anddirection information. Further, the images and/or LIDAR measureddistances may be associated to timing information. The timinginformation is provided with accuracy sufficient for the application.

The positioning system 511 comprises in one example a receiver of asatellite based positioning system, such as GPS. The positioning systemmay also comprise an inertial navigation system. The timing informationmay be provided from the receiver in the positioning system, saidreceiver being arranged to receive and process signals of a satellitebased positioning system, such as GPS.

Further, the arrangement 501 comprises a processing unit 512 arrangedto, based on the position and direction information related to the atleast one camera 504, stereo image process an arbitrary number of atleast partly overlapping image sets generated by the at least one cameraso as to provide the three dimensional model. In detail, the processingunit 512 is in one example arranged to find corresponding points in theat least partly overlapping images and to find disparity estimationsbased on the corresponding points so as to provide the stereo imageprocessing. In one example, the processing unit 512 is arranged to, foreach image to be stereo image processed, associate the position anddirection information so that each pixel of each image is correlated tocorresponding position and direction information. The stereo imageprocessing is then performed based on the associated position anddirection information so that all pixels in the 3D model are specifiedin three geographical dimensions.

In one example, the processing unit 512 is arranged to divide theenvironment into a plurality of areas or points, providing for each areaor point a plurality of geo-referenced image sets, wherein each imagecomprises the area or point, performing for each area or point imagestereo processing on each image set so as to provide a plurality of 3Dsub models for that area or point and providing the 3D model for eacharea or point based on the plurality of 3D sub-models. For example, the3D model for each area or point is provided by averaging the point orarea provided from the different sub-models. In one example, the imagesor image sets are associated to a weigh factor dependent on the qualityof the image. The averaging can then be weighted. Finally, theprocessing unit 512 is arranged to compose the 3D model based on the 3Dmodels related to the different areas or points.

The processing unit 512 may be arranged to perform bundle adjustment.

The processing unit 512 is further arranged to provide the 3D model alsobased on information from the distance measurement device 510. Theprocessing unit 512 is in one example arranged to develop the 3D modelbased on the plurality of overlapping images and to update the modelwith the distance information from the distance measuring device, whereappropriate. In detail, the 3D model covering a given area orenvironment is developed using the plurality of overlapping images. The3D model is then compared with information from the distance measuringdevice. In those parts of the environment where the 3D modelsubstantially coincides with the information provided from the distancemeasurements, the provided 3D model is regarded as finalized. However,in those parts of the 3D model where there is a discrepancy between the3D model and the information from the distance measurements, the 3Dmodel is updated in those parts, based on the distance information. Theweight of the distance information in the updated model may bedetermined as described above and will be exemplified more in detailbelow.

The processing unit 512 is arranged to verify that the 3D model partsupdated based on the distance information better describes the realityas presented in the images than the corresponding 3D model not updatedwith the distance information. This is in one example performed byre-projecting one image to another image in those parts where the modelhas been updated so as to determine if the updated model gives a betterre-projected estimated image than the not updated model. In detail, thenot updated model and the updated model can be determined based oncomparing an image I₂ taken from one certain location with differentestimated images Î₂ determined for the same certain location. Theestimated images are determined based on another image I₁ taken fromanother location and projected in the not updated 3D model respectivelythe updated 3D model to the position of the location of the image I₂.Thus the estimated image is determined as Î₂=f(I₁ M), wherein Mrepresents the not updated 3D model respectively the updated model. Incomparing the image I₂ taken from the certain location with theestimated images Î₂ that estimated image Î₂ which is most similar to theoriginal image I₂ is associated to the best model. In one example, theimages are compared or matched in small windows of the images. Thus, theimages are compared on a sub-image by sub-image basis. In one example,the matching is performed based on a correlation technique. In oneexample, the matching is performed based on a phase based algorithm. Inone example, the matching is performed based on a segmentation basedalgorithm.

Thus, if the model performed based only on the overlapping imagesprovides the best image estimate for the image I₂, then it is assumedthat the distance information should not be used in the model or atleast to a smaller extent than in the updated model. If on the otherhand, the model developed based also on the distance information givesthe best image estimate for the image I₂, then it is assumed that thedistance information enhances the 3D model.

The processing unit 512 may be arranged to verify in other ways known tothe person skilled in the art that the 3D model updated based on thedistance information better describes the reality as presented in theimages than the 3D model not updated with the distance information.

In those parts of the 3D model where it has been determined that thedistance information enhances the 3D model, the processing unit 512 isarranged to repeat the development of a model based on only overlappingimages. This time, the developing of the model is based on a selectionof the overlapping images in which the coordinates of the updated modelare visible. The processing unit 512 is then arranged to compare themodel with the distance information. If there is a difference betweenthe model and the distance information at some point, then the 3D modelis updated in this point as described above. The processing unit is thenarranged to verify that the 3D model updated based on the distanceinformation better describes the reality as presented in the images thanthe 3D model not updated with the distance information, as describedabove. If it is verified that the updated model better describes theenvironment, the processing unit 512 is arranged to repeat thedevelopment of the model based on a selection of the overlapping imagesand/or based on a selection of overlapping sub-images. If it is notverified that the updated model better describes the environment, theupdating of the model may be finalized or repeated again using adecreased influence from the distance measurement. If for example, thedistance information is wrong, such as if it has been measured against aflying bird, this distance information will not improve the model andwill thus be rejected in the verification of the updated model.

The processing unit 512 is in one example arranged to determine adifference between the developed model and a model developed in theprevious step and to exit the development of the 3D model if thedifference decreases a predetermined value. As is understood from above,the processing unit 512 may also be arranged to exit the development ofthe model if the distance information is determined not to improve themodel.

The processing unit 512 is in one example arranged to determine anuncertainty at some locations or at each location where LIDARinformation is available. The processing unit is then arranged todetermine the weight of the LIDAR measurement in the updated model basedon the uncertainty in the 3D model at that specific location. In oneexample, the uncertainty in the model is determined based on the anglebetween the optical axis of the camera images used for modelling and aplane of the surface of the model at that specific location. Forexample, for surfaces of the 3D model which are perpendicular to anoptical axis of the camera at that specific location, the uncertainty inthe 3D model is lower than for surfaces which are substantially parallelto the optical axis of the camera. Further, points measured by the LIDARdevice, which are not visible for any of the cameras since there aremodelled objects which obscure this location, the uncertainty of themodel may be regarded as high. The processing unit 512 may further bearranged to determine an uncertainty related to the LIDAR measurement.The weight of the LIDAR measurement in the updated model is thendetermined based on the uncertainty in the 3D model and/or theuncertainty in the LIDAR measurement.

In one example, the 3D model is represented as a mesh. In an alternativeexample, the 3D model is represented as a surface representation. In analternative example, the 3D model is represented as a voxelrepresentation.

The processing unit 512 comprises in one example a computer programcomprising a program code for developing a 3D model as discussed above.Further, a computer program product comprises a program code stored on acomputer readable media for developing a 3D model of an environment.

In the shown example, the arrangement 501 also comprises a memory 513for storing data related to the three dimensional model calculated bythe processing unit 512. The memory is also arranged to storeinformation related to the overlapping images and the distanceinformation. The arrangement 501 further comprises a display orpresentation unit 514 arranged to provide information related to the 3Dmodel. The display unit may be arranged to present a selected part ofthe 3D model. The arrangement 501 may also comprise input means (notshown) for selecting a part of the 3D model and the display is arrangedto present information related to the selected part.

The arrangement 501 may also comprise a transmitter (not shown) arrangedto transmit the information related to the 3D model to a receiver in aremote location. In one example, the transmitter at least partlysubstitutes the memory 513 and/or the display unit 514. In analternative example, the transmitter is provided in addition to thememory 513 and/or the display unit 514. In one example, the arrangementdoes not comprise the camera(s) 504 and the distance measuring device510. The processing unit 512 is then arranged to develop the model basedon image and distance information stored in the memory 513.

In FIG. 6, a first part of a procedure for developing a 3D model of anenvironment is illustrated. A 3D model 620 has been determined based onimages taken from a plurality of camera locations 604 a, 604 b, 604 c,604 d. In the illustrated example, a field of view 605 a, 605 b, 605 c,605 d is shown for the respective camera 604 a, 604 b, 604 c, 604 d atthe time of taking the images. Thus the field of view illustrates thecover of the images captured.

The 3D model 620 is compared to LIDAR measurement points 621 a, 621 b,621 c at corresponding locations. In the illustrated example, the 3Dmodel and the LIDAR measurements are substantially coinciding at a firstand third measurement location 621 a, 621 c. However, the LIDARmeasurement point 621 b differs from a corresponding point in the model620. In the illustrated example, the second measurement point 621 b ofthe LIDAR measurement is formed at the ground of an alley between twobuildings 622 a, 622 b. As the 3D model and the LIDAR measurementdiffers, the 3D model is locally drawn in a direction towards the LIDARmeasurement point. In one example, the 3D model 620 is drawn to at leastone updated point between the model and the second measurement point 621b as determined by a weight of the LIDAR measurement. In one example,the at least one updated point is selected such that the 3D model 620 isdrawn to the second measurement point 621 b. In one example, the atleast one updated point is determined based on the weight of the LIDARinformation. The weight of the LIDAR information is determined based ona predetermined scheme, as earlier discussed.

In one example the weight of the LIDAR measurement is also or insteaddetermined based on an uncertainty in the 3D model. In one example, theuncertainty in the model is based on the angular relationship betweenthe surface of the model 620 at the location of the second measurementpoint 621 b and the optical axes of the cameras when taking theavailable images. If the surface and the optical axes are close toparallel, then uncertainty is greater than if the relation between thesurface and the optical axes is close to perpendicular.

Further, in one example, the uncertainty in the model is instead or inaddition thereto determined by from how many or for how high percentageof the camera images the LIDAR measurement points are visible. In thisillustrated example, only a few of the camera images are taken from suchlocation and in such direction that the LIDAR measurement point would bevisible. The uncertainty is in one example then higher than if the LIDARpoint is visible from a high percentage or substantially all the imagesused in the development of the 3D model. The 3D model point isdetermined to be moved towards the corresponding LIDAR measurementlocation to an extent determined based on the 3D model uncertainty. InFIG. 7, a second part of a procedure for developing a 3D model of anenvironment is shown. An updated model 722 has been determined. Theupdated model is based on the model developed only using overlappingimage pairs and updated with at least one updated point 723 determinedbased on the distance information. As discussed above, the weight of thedistance information has been determined according to a predeterminedscheme The updated model 722 is evaluated. This was also described inrelation to FIG. 5. If the updated model 722 is considered to be betterthan the not updated model, then the model will be re-developed usingonly those images which shows the updated point(s) 723 of the 3D modelprovided based on the second measurement point (621 b in FIG. 6) andbased on the weight of the LIDAR measurement. In FIG. 7, a cone 724illustrates which images and/or sub-images may be used for there-developed of the model. The procedure as described in relation toFIG. 6 is repeated using the images in the determined cone. Thus, a 3Dmodel 620 will be determined based on the images taken from a pluralityof camera locations 704 b, 704 c within the cone 724. Sub-images mayalso be used, In the illustrated example, a field of view 705 b, 705 cis shown for the respective camera 704 b, 704 c at the time of takingthe images. Thus the field of view illustrates the cover of the imagescaptured.

In FIG. 8, it is illustrated an example wherein it would be useful thatweight of the LIDAR measurement is determined based on an uncertainty inthe LIDAR measurement. The pulse from a LIDAR device 810 used has a beamlobe 825 having a certain size in cross section. In the illustratedexample, one pulse for determining a distance generates two differentdistance results due to the extent of the lobe 825 in its cross section.Thus, there is an uncertainty in the distance measurements at sharpedges or boarders such as buildings. In this case, the weight of theLIDAR information may be less than in areas where a high spatialresolution is not required in a plane perpendicular to the beam.

In FIG. 9, a model of a forest is developed. In this situation, themodel produced by the images and the LIDAR measurements will providedifferent results. A model 920 provided based on overlapping images willbe close to the tree tops and potentially the tree tops will not bemodelled to their full extent. The LIDAR measurements will providespread measurement points. Some measurement points will be in the treetops and some measurement points will be close to ground. Thus situationcan be handled in a plurality of ways. In one example, the LIDARmeasurements are evaluated and it can be determined from the pattern ofpoints that these points cannot be used in the modelling. Then thoseLIDAR measurement points which are not desirable to use can be removed.In one additional or alternative example, the LIDAR measurement pointsare used. Those which are close to the ground will be determined not toimprove the model and will thus be rejected. Those LIDAR measurementpoints which are close to the tree tops can be used to model the treetops more close to their full extension. In a third example, the patternof the LIDAR measurement points are used for determining an uncertaintyin the LIDAR information and thus can be used in modelling to an extentdetermined by the uncertainty in the LIDAR measurement. These differentexamples may be combined.

In FIG. 10 an example of a method 1000 for developing a 3D model basedon overlapping images and distance information is illustrated. Themethod comprises providing 1010, overlapping images associated tonavigation data. Further, distance information is also provided 1020comprising a distance value and associated to navigation data.Thereafter, the 3D model is developed 1030 based on the plurality ofoverlapping images and the distance information. In developing the 3Dmodel, the 3D model is provided 1031 based on the plurality ofoverlapping images, the provided 1031 model is compared 1032 with thedistance information and the provided model is updated 1034 with thedistance information using an iterative process, where appropriate. Inone example, a weight relation between the distance measurement and theprovided 3D model is determined 1033. The 3D model can then be updated1034 with the distance information on the determined 1033 weight. Theweight relation may for example be determined based on a percentage of adifference between the value from the distance measurement and theprovided model. The weight relation can also or instead be determinedbased on an uncertainty in the provided 3D model. The weight relationmay instead or in addition thereto be determined based on an uncertaintyin the distance measurement.

In one example, the stereo distances are determined 1015 from theprovided overlapping images. The model can then be provided based on thedetermined stereo distances. In one example, the stereo distances aredetermined from the overlapping images based on correlation. In oneexample, the stereo distances are determining from the overlappingimages based on a phase based algorithm. In one example, the stereodistances are determined from the overlapping images is based onsegmentation based algorithm. In one example, any other stereo methodknown to the person skilled in the art is used.

The updated mode is then evaluated 1035 against the provided, notupdated model to determine which of the models is most accurate. If thenot updated model is determined to be most accurate, the development ofthe model may be finalized. If it is determined that the updated modelis most accurate, or at least it is determined that the distanceinformation does improve the model, the process is repeated. Theevaluation 1035 of the updated model against the provided modelcomprises in one example re-projecting the stereo images based on theupdated and the not updated 3D models and comparing the results.

The selection of images is then updated 1036 based on the coordinates tothe updated model. In one example, the updating 1036 of the selection ofoverlapping images comprises selecting only those images and/orsub-images which show parts of the environment relevant for the mostaccurate model (updated or not updated).

In one example the decision of repeating 1037 development of the modelcomprises the steps of determining a difference between the modelgenerated based on the overlapping images and a model generated based onthe selection of overlapping in a previous step. The development of the3D model can then be exited if the difference is below a selected level.In FIG. 11, a flow chart illustrating an example of a method 1100 fordeveloping a 3D model of an environment is shown.

In the illustrated example, a step of providing 1110 image datacomprises the steps of providing 1111 a plurality of overlapping imagesof the environment, providing navigation data 1112 and associating 1113the navigation data to the respective images. The navigation data maycomprise position information and a pointing direction of a cameracapturing the images at each instant of capturing. Thus, navigation datamay comprise information regarding position and orientation. It may alsocomprise information related to timing.

Further, as step of providing 1120 distance information comprises thesteps of transmitting 1121 a plurality of pulses such as laser from adistance measuring device for reflection in the environment, receiving1122 pulses reflected from the environment, providing 1123 navigationdata related to pulses and determining 1124 information related to arelation between time of transmission and time of reception of eachreceived pulse. The distance between the distance measuring device andthe reflection point associated to each pulse can then be determinedbased on the determined relation. Then navigation data is associated1125 to each determined distance. The navigation data may compriseposition information and the direction of the transmitted pulses. Thus,navigation data may comprise information regarding position andorientation. It may also comprise information related to timing.

In one example, the distance information is provided by means of LIDAR.In one example, the distance information is provided by means of radar.In one example, the distance information is provided by means of sonar.

Thereafter the 3D model is developed 1130 based on the provided imagedata and based on the provided distance information. In one example, theimage data and the distance measurements are provided using the sameplatform. It may then not be necessary to determine navigation dataspecifically to both the camera(s) and the distance measuring unit. Itmay then be enough to know the relation between the camera(s) and thedistance measuring device. The 3D model may be represented as a surfacerepresentation and/or a voxel and/or a mesh.

1. Method for developing a 3D model of an environment, comprising thesteps of: providing a plurality of overlapping images of theenvironment, each image being associated to navigation data; providingdistance information, said distance information comprising a distancevalue and navigation data from a plurality of distance measurements;developing the 3D model based on the plurality of overlapping images andthe distance information; wherein the step of developing the 3D modelcomprises the steps of providing the 3D model based on the plurality ofoverlapping images; and updating the 3D model with the distanceinformation using an iterative process, wherein the development of the3D model further comprises comparing the 3D model with information froma distance measuring device, updating the 3D model in parts of the 3Dmodel where there is a discrepancy between the 3D model and informationfrom the distance measurements, based on the distance information, andverifying that the 3D model parts updated based on the distanceinformation better describes the reality as presented in the images thanthe corresponding 3D model not updated with the distance information. 2.Method according to claim 1, further comprising a step of determining aweight relation between the distance information and the provided 3Dmodel, wherein the updating of the 3D model with the distanceinformation is based on the determined weight.
 3. Method according toclaim 2, wherein the weight relation is determined based on a percentageof a difference between the value from the distance measurement and theprovided model.
 4. Method according to claim 2, wherein the weightrelation is determined based on an uncertainty in the provided 3D model.5. Method according to claim 2, wherein the weight relation isdetermined based on an uncertainty in the distance measurement. 6.Method according to claim 2, wherein the step of developing the 3D modelcomprises the steps of: a) determining stereo distances from theoverlapping images; b) providing the 3D model based on the stereodistances from a selection of the overlapping images; c) determining thedifference between the 3D model and the distance information; d)updating the 3D model based on the difference between the 3D model andthe distance information and based on the determined weight relation, e)evaluating the updated model against the provided model to determinewhich of the models is most accurate f) updating the selection of theoverlapping images based on the evaluation, g) repeating step b) to f).7. Method according to claim 6, wherein step e) of evaluating theupdated model against the provided model comprises re-projecting thestereo images based on the updated 3D model.
 8. Method according toclaim 6, wherein the step f) of updating the selection of overlappingimages comprises selecting only those images and/or sub-images whichshow parts of the environment relevant for the most accurate model. 9.Method according to claim 6, wherein the step g) of repeating the stepof developing the 3D model comprises the steps of: determining adifference between the model generated based on stereo distances and amodel generated based on stereo distances in a previous step, whereinthe development of the 3D model is exited if the if the difference isbelow a selected level.
 10. Method according to claim 1, wherein thestep of providing of a plurality of overlapping images of theenvironment comprises: capturing a plurality of overlapping images ofthe environment using an imaging device; providing navigation datarelated to the images; and associating the plurality of overlappingimages with respective navigation data.
 11. Method according claim 1,wherein the step of providing the distance information comprises:transmitting a plurality of pulses from a distance measuring device forreflection in the environment; receiving pulses reflected from theenvironment; providing navigation data related to the transmitted and/orreceived pulses; determining information related to a relation betweentime of transmission and time of reception of each received pulse;associating positioning data to each information related the relationbetween time of transmission and time of reception with respectivenavigation data.
 12. Method according to claim 1, wherein the distanceinformation is provided by means of LIDAR, wherein the pulses are laserpulses.
 13. Method according to claim 1, wherein the distanceinformation is provided by means of radar.
 14. Method according to claim1, wherein the distance information is provided by means of sonar. 15.Method according to claim 1, wherein the navigation data comprisesinformation regarding position, orientation and optionally timing. 16.Method according to claim 1, wherein the 3D model is represented as amesh.
 17. Method according to claim 1, wherein the 3D model isrepresented as a surface representation.
 18. Method according to claim1, wherein the 3D model is represented as a voxel representation.